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Abstract 

This study is an examination of validity in published articles that have provided tests of the effectiveness of media literacy 
interventions. We identified 88 published tests of media literacy interventions then analyzed their content using five coding 
variables that indicated the degree to which authors of those studies established basic validity. We first conducted a meaning 
analysis to identify the definitions that authors of those studies presented for media literacy. Then we used those definitions 
to determine the extent to which those authors provided a complete (content validity) and accurate (face validity) operation¬ 
alization in the design of their measures. 


Highlights 

• The designs of published studies claiming to test media literacy interventions were analyzed to determine how the con¬ 
ceptual foundation was constructed in each study and whether the study was designed based on those conceptual founda¬ 
tions to determine content and face validity. 

• A total of 88 studies were selected after a series of electronic searches of studies that used the term “media literacy 
intervention” in their keyword lists, titles, and abstracts. 

• A meaning analysis found that 22 studies (25.0%) provided no conceptual foundation for media literacy, and 21 (23.9%) 
used an existing definition of media literacy. Despite there being hundreds of definitions for media literacy in the literature, 
the authors of the remaining 45 studies (51.1%) presented their own definition for media literacy. 

• The assessment of validity found that none of the studies presented a test of media literacy that completely captured the 
elements in their definitions of media literacy, so the content validity of this literature was judged as poor. 

• The evaluation of face validity uncovered many problems in a lack of correspondence between what authors intended to 
measure and what they actually measured. The most prevalent discrepancy was with measures of skills where authors 
frequently measured beliefs about study participants’ levels of skills rather than taking measures of actual performance. 

• We pose a series of three questions that illuminate the current nature of the media literacy intervention literature as well 
as serve to guide future designers of such studies. 
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When scholarly fields are brand new, researchers must 
make many assumptions about the nature of the phenome¬ 
non they are studying as well as the methods that might be 
useful in generating knowledge about that phenomenon. 
Then as a field’s research literature grows, scholars benefit 
from a larger base of knowledge, which allows them to use 
definitions of their key concepts that are more clear, com¬ 
plete, and insightful. And the literature provides more guid¬ 
ance about which research design elements (measures, 


treatments, samples, procedures, and analyses) are faulty and 
which are more useful. Therefore, a key indicator of the 
value of growth in a scholarly field is validity because valid¬ 
ity reflects the degree to which scholars provide clear defini¬ 
tions of a field’s key concepts as well as the degree to which 
scholars use those definitions to guide the design of their 
studies and the construction of their measures. 

In this study, we conduct a content analysis of published 
studies of media literacy interventions in order to generate 
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answers to two questions. First, when authors claim that 
their study provides a test of the effectiveness of a media 
literacy intervention, what do they mean by “media litera¬ 
cy?” We conducted a meaning analysis to generate an an¬ 
swer to this first question, then used the results of this 
meaning analysis to answer the second question: How well 
do designers of intervention studies use their conceptualiza¬ 
tions of media literacy to construct their measures? 

I. The Criterion of Validity in Social Science 
Research 

Validity is an essential concern for determining the 
quality of any social science research project (Brinberg & 
McGrath, 1985; Chaffee, 1991; Guilford, 1954; Nunnally, 
1967). Validity refers to the correspondence between what 
scholars say they are measuring and what they actually 
measure (Rust & Golombok, 1989; Vogt, 2005). Williams 
(1986) explains, “The question of validity is a question of 
‘goodness of fit’ between what the researcher has defined 
as a characteristic of a phenomenon and what he or she is 
reporting in the language of measurement” (p. 21). More 
concretely, validity is concerned with the match between 
the meaning of a concept and how it is operationalized in 
a research study (Chaffee, 1991). 

The task of assessing validity begins with a meaning 
analysis to determine how the authors of a study have de¬ 
fined their key concepts (Chaffee, 1991). This task proceeds 
by using authors’ meaning as the standard forjudging valid¬ 
ity. As Chaffee writes, “Validity should not be equated with 
‘truth.’ Disappointing as this might sound, the philosophi¬ 
cal concept of truth is not a usable criterion” (p. 11). Instead, 
the criterion forjudging validity within a research study is 
the meaning expressed by the authors of that study. This 
point is especially important in media literacy research 
where there are so many different conceptualizations of 
what it is. See Table 1 for a sampling of definitions in the 
literature, and notice that each definition has a different 
configuration of components (e.g., skills, knowledge, be¬ 
liefs). While there is considerable overlap among those 
components across definitions, each of those definitions is 
unique in terms of the configuration of those components. 

There are two general kinds of validity — the empirical 
type and the logical/conceptual type. The empirical type 
of validity relies on collecting data to show support for 
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expectations about what a particular concept should (predic¬ 
tive and concurrent) and should not (discriminant) be re¬ 
lated to, while the logical/conceptual type of validity is 
established through argumentation and expert judgments. 
In our content analysis, we focus attention on what are argu¬ 
ably the most fundamental logical/conceptual types of valid¬ 
ity - content validity and face validity. These types are 
fundamental because if these types cannot be established 
first, the testing for other types of validity has little meaning. 

A. Content Validity 

Content validity focuses attention on the structure of the 
meaning of concepts that is revealed when scholars provide 
definitions of those concepts. The assessment of content 
validity then entails the comparison of elements in that de¬ 
fined structure with the set of items that researchers use to 
measure that concept. Content validity is a judgment about 
the degree to which the set of measures used in a research 
study reflect all the elements laid out in the definition of a 
concept. Bausell (1986) illustrates the essence of content 
validity with the question, “Do the different components of 
the measurement procedure (which are usually items) match 
the different constituents of the attribute being measured?” 
(p. 156). Those items need to be representative of the entire 
concept (Vogt, 2005). Thus, content validity is stronger when 
designers of studies have used measures to assess all the ideas 
presented in their conceptualization. 

Across the many different definitions of media literacy 
in the literature, the two most prevalent definitional compo¬ 
nents appear to be skills and knowledge (see Table 2). How¬ 
ever, there are also frequent mentions of other components 
such as behaviors, affects, and beliefs. Notice that there is a 
considerable variety of ideas within each of these compo¬ 
nents. We use the term “domains” to refer to the different 
ideas within a component. For example, within the compo¬ 
nent of skills, some definitions call for the development of 
general skills, while other definitions focus on particular 
skills needed to access media messages, to interpret the 
meaning in those messages, and to produce messages - we 
consider these to be different “domains” of skills. Notice how 
the definitions presented in Table 2 display a variety both 
with components and with domains. Some scholars (e.g., 
Bachmair & Bazalgette, 2007; Hobbs, 1998) have claimed 
there has been a convergence towards a consensus definition 
where media literacy is defined as “the ability to access, 
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analyze, evaluate and communicate messages in a variety of 
forms” (Aufderheide, 1997, p. 79). This definition arose out 
of discussions of a group of scholars who convened the “Na¬ 
tional Leadership Conference on Media Literacy” in the 
mid-1990s. Such a definition is valuable because it provides 
designers of media literacy interventions with a widely 
shared conceptual foundation for their studies. However, 
other scholars are skeptical of this claim of a consensus 
definition for media literacy and point out that the field is 
characterized much more by its variety of definitions than 
its convergence to a consensus (Potter, 2010; Silverblatt, 
Ferry, Finan, 2015; Tyner, 2009). 

Our concern with validity in this study is much less on 
debating whether there is a consensus definition or not and 
much more on assessing how well designers of media litera¬ 
cy intervention studies operationalize their chosen defini¬ 
tions into measures. Thus, we begin our content analysis of 
published intervention studies with a meaning analysis to 
identify authors’ meanings they present for media literacy. 
We do not make judgments about the definitions themselves; 
instead, we focus attention on how well the authors use their 
selected definitions in the design of their studies. Therefore, 
the meaning analysis part of our content analysis is con¬ 
cerned with recording all the components and domains of the 
definitions for media literacy that authors present as the 
conceptual foundation of their empirical studies. 

Making a judgment of content validity involves compar¬ 
ing a study’s set of measures of media literacy against a 
criterion. In designing our content analysis study, we did not 
select one definition of media literacy as the criterion, be¬ 
cause there is no commonly accepted definition that could 
be used as a standard. Instead, we recorded what the authors 
of each study presented as their definition of media literacy 
and used the authors’ definitions as the standard to evaluate 
the degree of validity demonstrated in the construction of 
measures for that study. Thus for each study we examined, 
we conducted a meaning analysis to identify the components 
and domains expressed by the authors’ conceptualization of 
media literacy. We then compared this configuration of 
components and domains to the authors’ presented set of 
measures. The degree of content validity is indicated by the 
proportion of matches of the components and domains in 
authors’ conceptualizations of media literacy with the com¬ 
ponents and domains in their set of reported measures. 


B. Face Validity 

While content validity focuses attention on a concern 
about whether designers of media literacy studies have in¬ 
cluded measures for all the components and domains in their 
conceptualization of media literacy, face validity focuses 
attention on a concern about whether the measures used are 
acceptable operationalizations of the components and do¬ 
mains they purport to assess. If authors claim that a par¬ 
ticular measure is an assessment of component X but the 
measure is instead an assessment of component Y, then this 
non-match signals a barrier to establishing face validity. For 
example, let’s say that authors of an intervention study define 
media literacy as requiring the skill of critical thinking. In 
the methods section the authors say their measure of critical 
thinking consists of one item (/ am confident about my ability 
to think critically about media messages) and a five-point Likert 
type set of responses (1 = Strongly Agree, 2 = Agree, 3 = 
Neither Agree nor Disagree, 4 = Disagree, 5 = Strongly 
Disagree). When we compare this conceptualization to the 
measure, we can see a lack of correspondence, because the 
measure makes an assessment of respondents’ beliefs about 
the level of their skills rather than an assessment of their ac¬ 
tual levels. Instead, if the designers of the study defined media 
literacy as a condition where people have a high degree of 
confidence in their level of skill on critical thinking (regard¬ 
less of their actual level of skill), then this measure would 
correspond to the conceptualization and therefore qualify 
as exhibiting adequate face validity. 

II. Meaning Analysis of Media Literacy 

An analysis of authors’ meaning of their key concepts is 
essential as a first step in making judgments about content 
and face validity. Once authors’ meanings are found in their 
writings, those meanings become the standard for judging 
validity of measures as being complete (content validity) and 
accurate (face validity). 

There are a variety of ways that authors can convey their 
meaning for key concepts in their writing about empirical 
studies. One common way is for authors to assume that all 
readers share the same meaning for a term, so the authors 
treat it as a primitive concept and present no definition for 
the term in their writings. Chaffee (1991) explains that prim¬ 
itive terms are those where all people share the same meaning 
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so that it “would be foolish to expend a lot of effort on its 
definition” (p. 8). The most obvious primitive terms are ar¬ 
ticles (e.g., the, an) and prepositions (e.g., of, by, with) but 
they can also be nouns (e.g., person, chair, tree). In contrast 
to primitive terms are derived terms where authors recognize 
that there is not a common meaning, so authors need to 
present a definition to clarify for readers the meaning those 
authors are using in their study. This is especially important 
with hypothetical constructs (e.g., intelligence, attitude, 
anxiety) that might appear to be primitive terms but are 
actually technical terms that often vary in meaning across 
scholars. When these terms are used in scholarly forums, 
there is an obligation for authors to specify the definition 
they are using so readers can be clear about how they are 
being used. In his classic book Explication, Chaffee (1991) 
argues that scholars need to be more careful about avoiding 
assumptions that terms are primitive and instead carefully 
lay out the meaning for their readers. To guide this task, 
Chaffee (1991) presented a multi-step process he called ex¬ 
plication where scholars analyze literatures, make evalua¬ 
tions of the meanings that occur there, and move on to 
synthesize their own meanings that clarify trends in the 
literature and thereby articulate a clear foundation for their 
own studies. Chaffee’s multi-step process of explication sug¬ 
gests that there is a hierarchy of quality in the way authors 
of empirical studies can derive and present the meaning they 
are using for their focal concepts. We translate Chaffee’s 
procedure of explication into a six-level hierarchy that can 
be used to track the progress of a field’s progress towards 
precision over time (see Table 3). When a scholarly field is 
new, researchers can find little guidance in their literature 
for constructing a conceptual foundation for their studies 
and selecting measures with the highest degree of demon¬ 
strated validity, so their research designs make their contri¬ 
butions at the lower levels on this hierarchy. However, as the 
literature grows, research designers are provided with an 
increasing amount of guidance, so that they can base their 
design decisions less on untested assumptions and more on 
trusted patterns of empirical findings. Thus, movement up¬ 
wards on this hierarchy requires researchers to add more 
value as scholars by finding useful patterns in the literature 
then using those patterns to create conceptual foundations 
for their studies that are clearer and more precise. 

Level 1 on this hierarchy represents those studies where 
authors treat their focal concept - in this case, media literacy - 
as a primitive term. This option is regarded as exhibiting the 


lowest quality because it is based on the assumption that all 
scholars share the same meaning for a term. Given that the 
literature has grown large enough to exhibit many defini¬ 
tions of “media literacy,” research study designers are pro¬ 
vided with a great deal of guidance; thus, studies that 
operate at this level are regarded as faulty. 

Researchers who move their work upwards from Level 1 
recognize that they need to avoid the assumption that all 
readers share the same definition for media literacy, so they 
provide a definition they are using as the foundation for their 
study. If they simply present a definition, but they do not 
provide a source for the definition in the form of a citation 
in their review of the literature, then their work is limited to 
Level 2. Although their study demonstrates a higher level of 
precision compared to Level 1 by clearly communicating 
their definition to readers, they provide no scholarly founda¬ 
tion for that definition, so the presented definition appears 
arbitrary and is untethered to any previously published schol¬ 
arship on media literacy. Studies at this level leave the read¬ 
er wondering if the presented definition is in fact from the 
existing literature or whether the authors have constructed 
their own definition. If the latter, readers question why the 
authors created a new definition when so many already ex¬ 
ist. 

Researchers who move their work upwards from Level 2 
present a definition of media literacy and cite a source for it 
from the literature. Level 3 research has the advantage of 
providing readers with a clear articulation of the authors’ 
meaning and ties that meaning to a history of thinking 
through the citation. However, this option leaves readers 
wondering whether there were also other definitions consid¬ 
ered, and if so, why those other meanings were rejected. 

Researchers who move their work upwards from Level 3 
demonstrate an awareness that there are multiple meanings 
for media literacy in the scholarly literature. Level 4 research 
studies present a review of the literature that describes more 
than one definition of media literacy that concludes with the 
authors selecting one of those definitions as the foundation 
for their study. 

Researchers who move their work upwards from Level 4 
go beyond demonstrating an awareness of multiple defini¬ 
tions for media literacy and present their own constructed 
definition for media literacy. Level 5 research is superior to 
the other four lower level studies because it presents readers 
with a definition that appears to be synthesized through a 
critical evaluation of existing definitions. This synthesis 
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Table 3. Hierarchy of Quality in Conceptual Foundation, (back to pg. 5; forward to pg. 8) 

Level 

1 No Definition 22 studies (25.0%) 

Authors provide no definition of media literacy; media literacy is assumed to be a primitive term where all 
readers share the same meaning. 

2 Foundationless Definition 11 studies (12.5%) 

Authors present their study’s definition of media literacy with no foundation (i.e., there are no meanings for 
media literacy presented in their review of the literature). 

3 Selection from Single Definition 14 studies (15.9%) 

Authors present their study’s definition of media literacy with a foundation of citing a single meaning for 
media literacy in their review of literature; there is no acknowledgment that there are other meanings for 
media literacy. 

4 Selection from Multiple Definitions 7 studies (8.0%) 

Authors present their study’s definition of media literacy with a foundation that cites multiple meanings for 
media literacy in their review of literature; they select one of those meanings but do not explain why they 
selected that particular meaning for their study. 

5 Construction with No Explanation 26 studies (29.5%) 

Authors present their study’s definition of media literacy; although they cite multiple meanings for media 
literacy in their review of literature, they do not explain how they critically evaluated those multiple definitions 
in the construction of their study’s definition. 

6 Construction with Explanation 8 studies (9.1%) 

Authors present their study’s definition of media literacy with a foundation that cites multiple meanings for 
media literacy in their review of literature; they show how they critically analyze those definitions and 
construct their own definition from the results of their critical analysis. 


identifies the most useful elements in those definitions then 
assembles those definitional elements into a coherent defini¬ 
tion that is somehow superior to any of the existing options. 
However, the authors have not shown this process of critical 
evaluation and synthesis to readers. 

At the pinnacle of this hierarchy is Level 6, which dem¬ 
onstrates the full degree of explication suggested by Chaffee. 
Authors who publish Level 6 studies move beyond simply 
describing the multiple meanings in the scholarly literature 
and take the readers through the step-by-step process they 
used in critically analyzing the range of meanings in the 
literature and how those meanings were synthesized into a 
coherent set. Authors operating at Level 6 demonstrate con¬ 
siderable scholarly skill of synthesis in order to analyze 
meanings of a focal concept for their component elements, 
systematically make judgments about the value of all the 
elements in those definitions, then sort through the value of 


those elements so that the weakest are rejected, and the 
strongest are then assembled into a new configuration of 
meaning. While it might appear at first like the generation 
of another meaning would add to the definitional clutter, 
this is not the case. A good synthesized definition does not 
simply add another meaning; instead, it serves to replace 
much of the fuzzy definitional work with a definition that 
is not only more clear and useful but also builds from the 
strengths in the literature (Chaffee, 1991). 

When scholars conduct reviews of the literature, their 
task is made easier to the extent that the authors of the indi¬ 
vidual studies in that literature have operated at higher levels 
on this meaning explication hierarchy. Reviewers of the 
literature then can be more confident that they are identify¬ 
ing the actual meanings of those various authors have used 
in their studies rather than having to infer those meanings. 
Reviewers can then group those studies according to their 
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different meanings rather than treating all studies in a single 
group and thereby assuming that the authors across all those 
studies shared the same meaning. 

Grouping studies by meaning is also important when 
conducting a meta-analysis, which is a “review that uses a 
specific statistical technique for synthesizing the results of 
several studies into a single quantitative estimate (i.e., a 
summary effect size)” (Petticrew & Roberts, 2006, p. 19). 
Thus, when authors plan to conduct a meta-analytical re¬ 
view, a meaning analysis is a crucial first step in order to 
ensure that the studies selected for the sample all share the 
same meaning for their key concepts. One of the strongest 
criticisms of meta-analysis is that it “too often seeks to com¬ 
bine dissimilar studies — sometimes called the ‘apples and 
oranges’ problem” (Petticrew & Roberts, 2006, p. 203). It 
appears that Jeong, Cho, and Hwang (2012) were aware of 
the use of many synonyms for media literacy when they 
selected studies for their meta-analysis of the media literacy 
intervention literature. They said they started with four key¬ 
word phrases (media literacy, media literacy intervention, 
media literacy curriculum, and media literacy program) to 
select studies for their meta-analysis, then broadened their 
search criteria. “To include the studies that did not use the 
term ‘media literacy,’ we used search terms (e.g., ‘interven¬ 
tion,’ ‘advertising,’ ‘skepticism’)” (p. 466). This procedure 
indicates that Jeong and colleagues believed that these seven 
terms shared the same meaning and furthermore that the 
authors of the 51 studies they selected for their meta-analysis 
all shared this meaning. 

We believe this body of literature can benefit from a 
meaning analysis, and we present one in this manuscript. 
We use Chaffee’s (1991) hierarchy to structure our meaning 
analysis. Each article in the media literacy intervention lit¬ 
erature is analyzed to determine how authors have pre¬ 
sented their meaning for media literacy. We do not expect 
all empirical tests of media literacy to share the same mean¬ 
ing, nor do we expect all studies to be coded at the same 
option on the six-level meaning hierarchy. The distribution 
of studies across these six options will indicate the degree of 
precision authors of these studies exhibit in deriving mean¬ 
ing and presenting it to their readers. 

In summary, our content analysis of the empirical media 
literacy intervention literature is designed to document the 
meanings authors present for “media literacy” then to ana¬ 
lyze the measures those authors have used to operationalize 
their conceptualized meanings into measures. In our content 
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analysis, we record the presence of key characteristics of 
conceptualizations authors present for media literacy in 
those published studies to provide the foundation we use to 
make judgments about content validity and face validity. 
Specifically, we record the components and domains authors 
present in their definitions of media literacy that serve as the 
foundation for their studies. In assessing content validity, we 
compare the configuration of components and domains in 
the conceptual foundations with the set of measures pre¬ 
sented in each study. In assessing face validity, we make 
judgments about how well the measures the researchers use 
are adequate measures of the components and domains they 
say they are measuring. 

Method 

A. Selecting Media Literacy Intervention 
Studies 

Like Jeong, Cho, and Hwang (2012) did in their meta¬ 
analysis of media literacy intervention studies, we began 
with an electronic search of several communication litera¬ 
ture data bases (Communication Abstracts, Psyclit, and 
ERIC). But unlike Jeong and colleagues who used many 
keyword phrases (media literacy, media literacy intervention, 
media literacy curriculum, media literacy program, interven¬ 
tion, advertising, and skepticism), we used the single term 
“media literacy & intervention.” Readers may object to our 
not using additional search terms that they regard as syn¬ 
onyms for media literacy. However, we could not be sure 
which terms are synonyms in widespread use and which are 
believed to be synonyms by only a few scholars, so we de¬ 
cided to be conservative for the sake of precision. Admit¬ 
tedly, this resulted in a smaller set of studies than we would 
have generated by using more key-word terms, however, by 
using one key search term we attempted to avoid an “apples 
and oranges” non-equivalency. We argue that the fairest 
selection criterion was to rely on authors telling us whether 
or not their study was a test of a media literacy intervention 
by selecting their study only if they provided the key-word 
terms of both “media literacy” and “intervention.” 

When we identified a potential study, we searched 
through its reference list to identify additional studies that 
might have provided tests of media literacy interventions. 
We then screened out studies that were not published, 
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because published studies have gone through a scholarly 
review process and therefore have an expectation of higher 
quality than studies that have not. However, we screened out 
studies that did not provide a test of the effectiveness of in¬ 
terventions, because we wanted to see authors report their 
measures of media literacy so we could compare the mea¬ 
sures to their meaning in determining content and face valid¬ 
ity. 

Our resulting sample was 88 published articles that 
claimed to test the effectiveness of a media literacy interven¬ 
tion. All of these studies feature the terms “media literacy” 
and “intervention” prominently — either in title, keyword 
list, or abstract. Many of these studies featured these terms 
prominently in each of those places as well as many times 
in text. 

B. Coding Variables 

The development of the list of variables, their definitions, 
and their codes was a process involving many pilot tests over 
several years. When it was finished, the coding itself pro¬ 
gressed smoothly mainly because almost all of the coding 
variables were manifest, rather than latent, variables (Hol- 
sti, 1969; Krippendorff, 2012). That is, we minimized the 
need for coders to make their own inferences of authors’ 
meaning about media literacy by recording only those mean¬ 
ings expressed by the authors. 

The meaning analysis first involved the coding of three 
variables: Number of media literacy definitions cited, the 
authors’ focal definition of media literacy, and the compo¬ 
nents and domains in those focal definitions. When authors 
presented only one definition of media literacy, we used that 
meaning as an expression of their conceptual foundation. 
When authors presented more than one meaning for media 
literacy in their review of the literature, we looked for an 
expression by the authors indicating which of those defini¬ 
tions they were using as the conceptual foundation of their 
study. This procedure generated data on five variables that 
we describe below in more detail. 

Media literacy definitions cited. 

Coders counted the number of different definitions of 
media literacy that authors cited in their introduction, review 
of literature, and rationale sections. If a particular definition 
was presented more than once, it was counted only once. 
Also, if a particular definition was credited to more than one 


source, it was counted only once. 

Focal definition of media literacy. 

If authors cited only one definition in their review of the 
literature, coders regarded this one definition to be the au¬ 
thors’ conceptualization of media literacy. If the authors 
presented more than one definition of media literacy in their 
review of the literature, coders then looked for what the 
authors claimed to be using as the conceptual foundation for 
their study as well as an articulation of their reasoning pro¬ 
cess for their selection and rejections. Coders then catego¬ 
rized this conceptualization on one of the six levels of the 
meaning presentation hierarchy developed from Chaffee’s 
(1991) criteria (see Table 3). 

Definitional elements. 

Coders recorded the elements in the definitions used by 
authors as their conceptual foundation for media literacy. 
These elements took the form of components and domains. 
The definitional components were skills, knowledge, behav¬ 
iors, beliefs, attitudes, affects, and other. The domains were 
the more specific types of elements within each component. 
For example, in the skills component, domains were indi¬ 
vidual skills such as analysis, critical thinking, synthesis, 
etc. We did not begin with a list of all possible domains; 
instead we recorded what authors regarded as domains. 

We defined knowledge as acquired factual information. 
Factual information has truth value so that one can assess 
whether a fact is accurate or not. In contrast to factual in¬ 
formation is social information which is important but does 
not have a factual basis, so this type of information was 
coded either as a belief, an attitude, or an affect. 

Beliefs have been defined as cognitions about the prob¬ 
ability that an object or event is associated with a given at¬ 
tribute (Fishbein & Ajzen, 1975). Simply stated, a belief is 
faith that something is real or is true. 

Attitudes were defined as judgments about something. 
These evaluative judgments have valence and intensity (Fab- 
rigar, MacDonald, & Wegener, 2005). Valance refers to 
whether the object of the attitude meets (satisfactory), ex¬ 
ceeds (positive), or falls short of the standard (negative). In¬ 
tensity refers to how far from the standard the object is 
perceived to be. 

Affect refers to the feelings that people experience. Affect 
includes emotions and moods. Behaviors are typically de¬ 
fined as the overt actions of an individual (Albarracin, 
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Zanna, Johnson, & Kumkale, 2005). 

Skills were defined as cognitive abilities that can be de¬ 
veloped through training and practice. Wiley (1991) writes 
that skills “are abilities to perform tasks. Most such abilities 
are acquired, i.e., learned” (p. 78). He says that the valid 
measurement of skills requires observation of how people 
perform on a task, which is “a goal-oriented activity of de¬ 
terminable duration on which performance can be evaluat¬ 
ed” (p. 105). While the measurement of skills requires the 
observation of performance, oftentimes researchers will 
measure something (like a belief or attitude) and argue that 
this measure is a valid outcome of the application of a skill. 
Such a procedure is a shortcut that requires a strong argu¬ 
ment to convince readers that such an outcome could only 
have occurred by the use of the skill in question. This is an 
especially important argument to make convincingly in 
media literacy intervention studies because the purpose of 
the intervention is typically to increase the level of a par¬ 
ticular skill or set of skills that the authors regard as the 
media literacy part of the intervention. In some studies, the 
skill was the outcome variable, that is, the participants who 
experienced the media literacy intervention were expected 
to exhibit higher scores on the skill measure compared to 
participants who did not experience the media literacy inter¬ 
vention. In other studies, the authors exhibited more interest 
in another outcome variable, such as an attitude or belief 
about some risky behavior; these studies had a two-step 
structure where the media literacy intervention was ex¬ 
pected to increase skill levels and those elevated skill levels 
were expected to explain changes on the outcome variable. 

When authors specified a component, coders also re¬ 
corded whether authors specified domains within that com¬ 
ponent. For example, if authors defined media literacy in 
terms of skills, then coders looked for whether authors ar¬ 
ticulated particular skills (such as analysis, evaluation, read¬ 
ing, message production, etc.). If authors defined media 
literacy in terms of knowledge, then coders looked for wheth¬ 
er the authors specified types of knowledge (such as how the 
industry works, content formulas, etc.). 

Content validity. 

Coders did not make a global judgment about content 
validity but instead used a system of counting components 
and domains. First, coders examined authors’ conceptualiza¬ 
tion of media literacy and recorded the components and 
domains in those definitions. For example, let’s say that the 


authors of a media literacy intervention study defined media 
literacy as knowledge about the motives of media companies, 
the skills of analysis and evaluation, and attitudes about the 
media. In this case, coders would record three components 
(knowledge, skills, attitudes) and four domains (one domain 
in the knowledge component, two in the skills component, 
and one in the attitude component). 

Second, coders counted the number of components for 
which the authors provided measures. Returning to the ex¬ 
ample, if the authors listed measures that tested their par¬ 
ticipants’ knowledge of the motives of media companies, 
their skill of analysis, their skill of evaluation, and their at¬ 
titudes about the media, then coders would list four compo¬ 
nent/domains measured. This would be a match of 100% 
and indicate perfect content validity. However, if authors 
presented measures for only three of these component/do¬ 
mains, then coders would record those three which would 
indicate 75% on content validity. 

In a few studies, the authors presented a conceptualiza¬ 
tion of media literacy that had many components and do¬ 
mains but then argued that it was too much for one study to 
test all of these components and domains. If those authors 
articulated that only a sub-set would be tested in their study, 
then we used the sub-set as the criteria for judging content 
validity and face validity. 

Face validity. 

Coders made a judgment about whether or not each 
measure of media literacy matched the component-domain 
unit that the authors claimed. Thus we did not try to make 
judgments about how good the measures were on a contin¬ 
uum; instead, we limited ourselves to counting the number 
of matches. For example, if authors expressed a component 
of knowledge and a domain of knowledge about media in¬ 
dustries, then we looked for measures testing recall of facts 
about the media industry that were taught in the interven¬ 
tion; we did not make a judgment about how complete those 
measures were as tests of all aspects of media industries, nor 
did we make a judgment about whether those facts were core 
or peripheral, nor did we make a judgment about how clear¬ 
ly worded those measures were - all of which would have 
required a considerable degree of judgment from coders. We 
focused only on fit, that is, did the measure fit the component 
and domain that the authors claimed? Therefore, if authors 
claimed that a measure such as “How many hours during 
an average week do you read business publications about the 


www. rcommumcationr. org 


46 



Reviewing Media Literacy Intervention Studies for Validity 


media industries?” was a measure of knowledge, we made 
a judgment that this measure did not fit their conceptualiza¬ 
tion because it was a measure of exposure to information, 
not a measure of how much knowledge the participants ac¬ 
quired and retained. 

C. Reliability 

The two authors served as coders on the project. Both 
coders coded 55 articles thus creating an overlap of 22 arti¬ 
cles that was used to test for inter-coder reliability. The 
percentages of agreement were corrected by Scott’s pi are as 
follows: definitions cited, .94; position on meaning hierarchy, 
.89; definitional elements, .84; measures of media literacy, 
.82; and face validity, .79. 

Results 

Presentation of Meaning 

Across these 88 studies, there was a considerable variety 
in the way authors presented their meaning of media litera¬ 
cy (see Table 3). Authors of 22 studies (25%) provided no 
definition for media literacy, which indicates that they were 
treating it like a primitive concept with the assumption that 
all their readers would share the same meaning. 

The 66 (75.0%) articles that did present definitions of 
media literacy in their review of the literature exhibited a 
wide variation. The authors of 11 studies (12.5%) presented 
a definition without a citation as to the source of that defini¬ 
tion and no review of the media literacy definitions litera¬ 
ture. Thus, there were 55 studies that presented one or more 
definitions of media literacy with citations that indicated the 
source of those definitions. Within these 55 studies, the 
range was 1 to 7 definitions cited with a median of 2 defini¬ 
tions. It is interesting to note that across these 55 studies, the 
most prevalent definition, which was the NAMLE defini¬ 
tion, was mentioned only 7 times (12.7%), which shows that 
there is no consensus definition for media literacy, at least 
among designers of published media literacy intervention 
studies. 

Authors of 14 studies (15.9%) presented only one defini¬ 
tion of media literacy in their review of the literature and 
used that definition as a foundation for their study, while 


authors of 7 studies (8.0%) presented multiple definitions of 
media literacy in their reviews and selected one of these 
definitions as their foundation. 

While the authors of the remaining 34 studies presented 
a review of the literature showing multiple meanings of 
media literacy and rejected all the definitions preferring to 
construct their own definition, the minority of these (8 stud¬ 
ies) displayed a scholarly treatment of that constructed defi¬ 
nition as suggested by Chaffee. That is, in 8 studies the 
authors displayed multiple definitions and furthermore crit¬ 
ically analyzed those meanings to show readers how they 
constructed their own definitions; in the other 26 studies, 
the authors simply described the multiple meanings then 
presumably rejected them all by presenting another defini¬ 
tion of their own. 

The most popular component to be mentioned in the 
conceptual foundations was some sort skill; 59 of the 66 
studies mentioned the component of skills in their definition 
of media literacy. The next most prevalent component men¬ 
tioned was some knowledge, then there is a drop off in 
counts to behaviors, beliefs, and attitudes (see Table 4). 

In summary, our meaning analysis revealed a wide dis¬ 
tribution across all six levels of the meaning presentation 
hierarchy. Over half the studies were categorized at level 3 
or lower. 

Validity 

In making our judgments about validity, we considered 
only the 66 articles that presented a definition for media 
literacy; the other 22 articles were ignored because there was 
no basis for making a judgment about validity. Also, if the 
authors of a particular study presented a definition with 
multiple components/domains but then said they were lim¬ 
iting their study to test only a sub-set of these, we used only 
the specified sub-set as the standard for validity. But if au¬ 
thors did not say they were limiting their test to a sub-set, we 
used the full set of components/domains as a standard for 
validity. 

The results of our validity analyses are displayed in 
Table 4. Studies that presented a definition of media literacy 
that included a skill component are presented on the first 
line of the table. If the authors defined media literacy in 
terms of a knowledge component, it appears as a count on 
the second line. If a study’s conceptual foundation called for 
more than one component, it was counted more than once 
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and therefore shows up more than once in Table 4. 

When we look at the general pattern exhibited in Table 
4, we can see that there are considerable discrepancies be¬ 
tween the components authors specified in their definitions 
of media literacy compared to the measures they reported. 
Some of these discrepancies show up as drop-offs, that is, 
authors define media literacy with more components than 
they measure. For example, with the component of skill, 59 
studies included this component in their definition but only 
22 of those studies presented a measure for any type of skill. 
This pattern of a drop-off is also seen with the component 
of affect. 

On other components, the discrepancy is not a drop-off 
but an increase, that is, there are higher counts in the mea¬ 
sured column compared to the counts in the conceptual 
foundation column. For example, in 11 articles the authors 
argued that beliefs were an important component in their 
definition of media literacy and yet 24 measured beliefs as 
part of media literacy. At first, this might seem like a pattern 
of over-measurement where authors establish no basis for 
beliefs in their definition but then develop a measure for it. 
However, this is not the explanation for the pattern. Instead, 
this pattern is explained by authors attempting to measure 
one type of component but then designing items that mea¬ 
sure another type of component. With the component of 
beliefs, the reason the number in the Measured column is 


much larger than the number in the Definition column is 
because many researchers who were attempting to measure 
a skill developed items to measure a belief instead. For ex¬ 
ample, several researchers defined media literacy as the 
ability to analyze media messages but then, rather than de¬ 
sign a measure to assess their participants’ ability to analyze, 
they instead asked their participants to respond to a state¬ 
ment (e.g., I am confident in my ability to analyze media 
messages) by choosing a number from a five-point Likert 
type scale (e.g., strongly agree, agree, neutral, disagree, 
strongly disagree). By doing so, they end up measuring par¬ 
ticipants’ beliefs about their skills rather than measuring par¬ 
ticipants’ level of performance on their skills. Another 
disconnect with measuring skill occurred when researchers 
asked their participants to respond to a statement (e.g., How 
often do you analyze characters when you watch television 
shows?) by choosing a number from a five-point Likert type 
scale (e.g., always, almost always, often, rarely, never). Here, 
they were measuring participants’ self-reported behaviors 
rather than the level of performance on the skill. 

When we shift attention from components to domains, 
we dig deeper into this pattern of disconnects between con¬ 
ceptualizations and operationalizations. Table 5 shows that 
the skills component was not only the most prevalent in the 
conceptual foundations, but this component displayed the 
most domains with 29 studies specifying one particular skill, 


Table 4. Analysis by Components for Media Literacy by Studies, (back to pg. 10) 


Component 

In Definition 

Measured 

Matches 

Skills 

59 

22 

12 

Knowledge 

25 

26 

9 

Behaviors 

12 

16 

3 

Beliefs 

11 

24 

7 

Attitudes 

9 

22 

7 

Affects 

5 

1 

0 

Other 

6 

7 

2 


n = 66 studies. Numbers in the cells represent studies. 

The numbers in the columns do not sum to 66 because almost all studies referred to more than one 
component. 
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13 studies specifying two particular skills, 5 studies specify¬ 
ing three skills, and 9 studies specifying four or more skills. 
When we compare the number of mentions of skill domains 
in the conceptual foundations with which skills domains 
that were measured in those studies, we can see a big drop¬ 
off. This drop-off is also exhibited in the other five compo¬ 
nent areas, but it is not as pronounced as with the drop-off 
in the measurement of skills. 

When we look for matches within each study, we can see 
that the drop-offs are even more pronounced. Table 6 pres¬ 
ents a comparison between how many domains are specified 
in the conceptual foundations and how many of those do¬ 
mains were measured. Within the skills component, 30 
studies specified one skill domain in the conceptual founda¬ 
tion, but only 5 of those studies provided a measure for that 
particular domain, and 13 studies specified two skill do¬ 
mains but only 3 of those studies provided measures for each 
of those two. 

V. Discussion 

We begin this discussion by acknowledging that all re¬ 
search studies have flaws; there are no perfect studies. By 


pointing out the flaws in the media literacy intervention 
literature, we are not advocating perfection as a standard; 
instead, we are arguing for a standard that we believe is not 
only reasonable but one that is also essential if this literature 
is to grow in value. In this Discussion section, we will first 
present that standard and show why it is reasonable and es¬ 
sential. Then we will raise some important questions about 
the design of media literacy intervention studies - questions 
that we will examine in the second half of this Discussion 
section. 

A. Reasonable Standard 

When authors present their study as a test of a media 
literacy intervention, we recommend they create a concep¬ 
tual foundation that clearly communicates to the reader what 
their meaning is for media literacy, then use this meaning to 
guide their selection and/or construction of measures of 
media literacy. Thus we argue that in order for a study to be 
considered a test of a media literacy intervention it must meet 
two minimum criteria. First, authors must present an ar¬ 
ticulation of what they mean by media literacy as a concep¬ 
tual foundation for their study. Second, authors need to 
present measures of media literacy that conform to their 


Table 5. Analysis by Domains for Media Literacy by Studies, (back to pg. 11) 


# of Mentions in Definition 

Skills 

Know 

Beh 

Belief 

Att 

Affect 


One 

29 

23 

9 

12 

7 

8 


Two 

13 

5 

5 

1 

1 

0 


Three 

5 

3 

1 

1 

1 

0 


Four or More 

9 

6 

0 

0 

0 

0 


Totals 

56 

37 

15 

14 

9 

8 

139 

# of Mentions in Definition 

Skills 

Know 

Beh 

Belief 

Att 

Affect 


One 

12 

13 

9 

10 

11 

0 


Two 

6 

3 

2 

3 

3 

1 


Three 

1 

2 

0 

3 

2 

0 


Four or More 


1 

4 

1 

3 

0 

0 

Totals 

20 

22 

12 

19 

16 

1 

105 


n = 66 studies. Numbers in the cells represent studies 
Note: Know = Knowledge; Beh = Behavior; Att = Attitudes 
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conceptual foundation. 

We argue that this is a reasonable standard because schol¬ 
ars who search for studies about media literacy interventions 
expect to learn something about how media literacy can be 
taught to others and how that teaching can be most effective. 
Therefore if authors label their work as a media literacy in¬ 
tervention study by using the term in the title, abstract, 
keyword list, and throughout their article, then they mislead 
readers when they do not tell them their meaning for media 
literacy. 

Conceptual foundation. 

This recommendation might seem so obvious that there 
would be no need to state it. However, our findings revealed 
that 25% of the studies in our sample provided no foundation 
for media literacy, which is shocking because this is an easy 
criterion to meet. 

In making this recommendation that authors of media 
literacy intervention studies tell readers what their meaning 
of the term is, we are not arguing for the use of any particu¬ 
lar meaning. We believe that the many definitions of media 
literacy already displayed in the literature is a strength be¬ 
cause it offers many different choices of meanings for 


scholars at this stage in the development of the field. How¬ 
ever, given this rich resource of alternative meanings, au¬ 
thors cannot assume that all their readers will share the same 
meaning for the term as their authors do, so we argue that it 
is imperative for authors to communicate their preferred 
meaning. When authors ignore this task, they fail to provide 
a conceptual foundation that would increase the contribution 
their study can make to knowledge about media literacy. 

Authors can easily meet this standard by simply writing 
a sentence in their manuscripts where they provide their 
definition of media literacy. Of course, it would be better if 
authors also presented a review of the media literacy litera¬ 
ture to demonstrate an awareness of different definitions to 
give their work a more scholarly grounding. Moreover, it 
would be better still if authors presented a critical analysis 
of that literature that would show readers either why they 
selected one of those definitions over others or why they 
constructed their own definition by synthesizing the best 
definitional elements in that literature so that their concep¬ 
tual foundation would have maximum value as a scholarly 
foundation for their particular study. 

This recommendation that authors clearly articulate their 
meaning for media literacy is also essential because without 


Table 6. Analysis of Domains of Skills and Knowledge, (back to pg. 12) 


Skills 

Domains in CF 

Zero 

1 

2 

Number of Domains Measured 

3 4 5 

6 

Percent 

1 

25 

5 






16.7% 

2 

7 

3 

3 





18.8 

3 

1 

4 

0 

0 




0 

4 

2 

0 

2 

1 

0 



0 

5 

2 

0 

1 

0 

0 

1 


25.0 

Knowledge 









Domains in CF 

Zero 

1 

2 

3 

4 

5 

6 

Percent 

1 

5 

5 






50.0% 

2 

2 

2 

1 





20.0 

3 

1 

0 

1 

1 




33.3 

4 

0 

2 

0 

0 

0 



0 

5 

0 

0 

0 

0 

0 

1 


100.0 

6 

1 

0 

1 

0 

0 

0 

1 

100.0 


Note: CF = Conceptual Foundation 
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this, there is no basis for readers to judge the validity of 
measures. 

Using valid measures is an essential requirement of all 
social science empirical research (Croucher & Cronn-Mills, 
2015; Merrigan & Huston, 2008). 

Measures. 

The measures of media literacy included in a study can 
be used in a variety of ways, such as an outcome variable, 
an intervening variable, or an antecedent variable. As an 
outcome variable, media literacy measures can be designed 
as a post-test administered after an intervention. The expec¬ 
tation then is that this measure would be an assessment of 
whether the intervention had an immediate influence on 
participants’ levels of media literacy. However, in order to 
test this expectation, designers also need to use the measure 
in a pre-test. Unless they administer the measure of media 
literacy both before and after the intervention, researchers 
have no way of telling if the intervention changed partici¬ 
pants’ levels of media literacy. As an intervening variable, 
measures of media literacy can be gathered after an interven¬ 
tion then used to predict some outcome variable like the 
counter-reading of media messages that advocate risk behav¬ 
iors, such as smoking, illegal drug consumption, or unpro¬ 
tected sex. As an antecedent variable, measures of media 
literacy can be administered before a treatment then used as 
a predictor of the success of the intervention. Whatever their 
position in a research design, media literacy measures are 
essential when researchers position their study as a media 
literacy intervention study. 

In our sample, we found 21 studies (23.9%) that pre¬ 
sented no measures of media literacy. Of these 21 studies, 18 
also provided no definition of media literacy while the other 
three studies did provide a definition but did not measure 
media literacy. Although 66 studies presented measures of 
media literacy, most of these studies failed to craft measures 
that measured the components/domains they explicitly cited 
in their definitions. For example, 59 studies defined media 
literacy at least in part through skills but only 22 (37.3%) of 
these studies claimed to provide a measure of that skill in 
their Methods section, and only 12 of those 22 studies pro¬ 
vided a measure of skill rather than a measure of a belief 
about skills. If we look down the “Matches” column in Table 
4, we can see that there were 40 measures that had face valid¬ 
ity, that is, where authors presented a conceptual foundation 
for media literacy, constructed a measure to assess a domain 


in that definition, and actually measured that domain. If 
these 40 valid measures were spread out over 40 different 
studies, this would indicate that 24 studies (66 - 40) that 
provided a conceptual definition of media literacy but did 
not measure any of the domains in their definition. 

When we add these 24 studies that do not meet the sec¬ 
ond of our criteria to the 22 studies that do not meet the first 
criterion, we can see that half the studies in our sample do 
not meet both criteria and should therefore not be considered 
as part of the media literacy intervention literature despite 
the authors of those studies labeling them as such. We are 
not arguing that these studies fail to contribute something 
valuable to the larger media effects literature. Instead, our 
argument is that authors mislead readers and searches of the 
literature when they label their studies as tests of media lit¬ 
eracy interventions but then fail to present what they mean 
by media literacy and/or fail to present any measures of 
media literacy. This is a serious flaw in the media literacy 
intervention literature that can be corrected relatively easily 
in the future. 

B. Important Questions 

Our analysis of the current literature on media literacy 
interventions raises many questions about why those pub¬ 
lished authors have made the design decisions the way they 
did. We conclude this Discussion section with the posing of 
three of these questions, which form a sequence of issues that 
researchers must deal with when they design any test of a 
media literacy intervention. We will show that if authors 
confront the issues posed by this sequence of questions rath¬ 
er than ignore them, they will put themselves in a position 
to design tests with stronger validity and hence their findings 
will contribute more value to the media literacy intervention 
literature. 

Basic research or evaluation study? 

Are researchers more interested in (a) determining the 
overall performance of the intervention or (b) isolating one 
factor (or a small number of factors) in the intervention to 
determine whether it (they) influences the outcome variable? 
At first glance, this might seem to be a subtle difference, but 
as we will show in this section, the difference presents a fork 
in the road for a series of design decisions that would result 
in very different types of studies being conducted, and there¬ 
fore very different sets of results being generated (Coffman, 
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2003; Chen, 2013; Penfield, Baker, Scoble & Wykes, 2014; 
Reinking & Alvermann, 2005; Stufflebeam, 2007). The 
decision of which path to take can be guided by a consider¬ 
ation of seven characteristics: (a) who designs the interven¬ 
tion, (b) complexity of the intervention, (c) nature of targets 
of the intervention, (d) nature of the agents who will deliver 
the intervention, (e) standard for judging success of the in¬ 
tervention, (f) pilot testing, and (g) ultimate goal for testing 
the intervention. 

Evaluation study. Designing an evaluation study is more 
useful than designing a basic research study when scholars 
are presented with seven characteristics. First, the interven¬ 
tion is designed by a sponsoring agency or people hired or 
funded by the sponsoring agency (such as a public school 
system, a health agency, a consumer activist group, or a 
philanthropic institution). Second, the intervention is typi¬ 
cally a combination of many presentation elements (such as 
lectures, print materials, videos, and websites) that involve 
targets in many activities (e.g., watching, reading, critiquing, 
discussing, and producing). Evaluation researchers assume 
that all of these elements work together in a system as the 
intervention is delivered in a series of lessons spread out over 
time. Third, the intervention is administered to intact groups 
in the field (e.g., such as elementary school classrooms) 
where random assignment of targets to conditions is impos¬ 
sible and where random assignment of intact groups to con¬ 
ditions is typically limited. Fourth, the people who deliver 
the intervention are typically part of the naturalistic environ¬ 
ment (e.g., elementary school teachers, parents) who are 
given some training to administer the intervention but who 
are not expected to be perfectly matched on all instruc¬ 
tional criteria, which introduces unavoidable differences in 
the way the intervention is administered. Fifth, the agency 
has created standards for success of the intervention before 
its administration. Sixth, there are typically several rounds 
of pilot testing to help improve the success of the intervention 
where improvements focus on increasing clarity of materials. 
And seventh, the ultimate goal of this process is to develop 
an instructional package that can be disseminated to other 
groups in the hope of overcoming some widespread problem 
or trying to make society better in some way. 

Basic research study. In contrast, designing a basic research 
study is more appropriate to the extent to which the media 
literacy intervention meets the following seven characteris¬ 
tics. First, the intervention is designed by the scholars — not 
a sponsoring agency - who also design and execute the test. 


Second, researchers focus on only one element in the inter¬ 
vention in an effort to isolate its singular influence on the 
outcome variable. In order to do this, researchers try to 
control all the factors in the intervention and vary only one 
element across different treatments. Third, the intervention 
is administered as an experimental treatment to volunteers 
who typically come to a laboratory setting where their ex¬ 
perience with the intervention can be highly controlled to 
ensure that all participants in a treatment group are given 
the same experience. Fourth, the intervention is adminis¬ 
tered by the researchers or their confederates in a controlled 
situation so that uniformity is maintained across all partici¬ 
pants and treatment conditions with the exception that one 
of the factors of influence is carefully varied across each 
treatment condition. Fifth, there are no a priori standards of 
success but instead, researchers compare outcome variable 
means across the different treatment groups. Sixth, pilot 
testing is used to improve the distinctiveness of the factors of 
influence that are varied across treatments rather than trying 
to increase the value of the treatment to all participants. And 
seventh, the ultimate goal of this process is to make claims 
about the relative strength of different factors of influence. 

Within the published literature of media literacy interven¬ 
tion studies, it appears that all of the studies in our sample 
exhibited a basic research design although 37 of those stud¬ 
ies reported a situation that had more matches (on these 7 
characteristics) with the evaluation option compared to the 
basic research design option. However, none of the 37 were 
a complete match on all 7 evaluation elements with none of 
the 37 reporting that there was an a priori established stan¬ 
dard for effectiveness. One of the key characteristics of eval¬ 
uation studies is the use of benchmarks that are determined 
by the agency before the design of the study (McKenzie & 
Smeltzer, 1997). These benchmarks are then used to deter¬ 
mine whether the intervention was successful or not. These 
benchmarks are laid out a priori by the funding agency or 
by the goals of the curriculum within which the intervention 
is being tried. An example of such a benchmark is: After 
experiencing the intervention lesson that teaches about me¬ 
dia industries, 70% of all children will provide correct an¬ 
swers on at least 8 items on the 10-item post-test of knowledge 
about media industries. 

None of the studies in the analysis presented any bench¬ 
marks to be used as criteria for effectiveness. Instead, all 37 
studies defaulted to looking for statistical differences across 
treatment and control groups. Because none of these 


www.rcommumcationr.org 


52 



Reviewing Media Literacy Intervention Studies for Validity 


evaluation studies compared performance to a standard, 
none were able to make meaningful claims about the effec¬ 
tiveness of their tested interventions; instead, they were 
limited to making statements about whether there was a 
statistical difference in means between the group of partici¬ 
pants who experienced the intervention compared to the 
participants who did not (the control group). By largely fol¬ 
lowing many but not all of the conventions of evaluation 
studies, these studies were hybrids. 

We are not arguing that hybrid designs are always weak¬ 
er than “pure” designs; to the contrary, hybrid studies can 
be stronger than traditional designs if scholars synthesize 
the strongest elements of each option into a single design 
that amplifies strengths and minimizes weaknesses. How¬ 
ever, if the hybrid design arises from designers taking short¬ 
cuts (e.g., it is easier to test for group means than to develop 
adequate performance criteria), then the hybrid fails to take 
advantage of inherent strengths of a particular method. Such 
hybrid designs can still make a contribution to literature, but 
the value of that contribution is limited (or enhanced) by the 
degree to which researchers reduce the design weaknesses 
by increasing strengths. The pattern of design decisions with 
evaluation studies that we found indicates that designers 
have been eliminating a strength (use of standards for suc¬ 
cess) and replacing it with a weakness (comparison of 
means). 

We recommend that designers of media literacy interven¬ 
tion studies carefully consider the strengths and weaknesses 
of both basic research and evaluation studies. Designers can 
begin this task by thinking about the extent to which their 
envisioned study meets the 7 characteristics described above. 
For example, if their planned study meets more characteris¬ 
tics of an evaluation study than a basic research study, de¬ 
signers should consider whether they can change the 
remaining characteristics in their situation to meet the needs 
of an evaluation study more completely. One way to do this 
would be to create performance criteria as a benchmark of 
success. The analysis, then, could avoid comparing group 
means and instead compare the performance of individuals 
to the standard and report how many participants met the 
standard after experiencing each of the tested interventions. 

Media literacy as persuasion or empowerment? Do authors 
regard the purpose of the intervention as persuading people 
to change their beliefs (or behaviors) from something the 
authors regard as faulty to something else the authors regard 
as desirable? Or instead, do authors regard the purpose of 


the intervention to empower people to make better decisions 
on their own? This distinction reveals a key element in many 
conceptualizations of media literacy, which is empowering 
individuals to think for themselves and not automatically 
accept the meanings presented in many media messages. The 
choice here makes a big difference in how the interventions 
are designed and what is regarded as a standard to use in 
judging the success of the intervention. 

Researchers who take the persuasion path of thinking, 
design the intervention to convince people to reject one belief 
(or behavior) and replace it with another one. In order to 
achieve this goal, researchers design an intervention that 
contrasts the faulty nature of a belief that participants are 
assumed to hold with an alternative belief that the research¬ 
ers regard as better in some way. Success is then measured 
by examining how many participants exhibited the research¬ 
er-sanctioned belief following the intervention. Thus, data 
analysts look for convergence as evidence of effectiveness, 
that is, the extent to which all participants accepted the al¬ 
ternative belief after experiencing the intervention. The more 
participants who converge to accepted this belief, the more 
effective the intervention is judged to be. 

Researchers who take the empowerment path, in con¬ 
trast, design an intervention to help participants learn how 
to analyze media messages so they can determine for them¬ 
selves which meanings are faulty or not useful for their own 
purposes. Such interventions are designed to show partici¬ 
pants the risks of automatically processing media messages 
and show them alternatives to this automatic processing so 
that those participants learn to do something that can be 
applied in their everyday lives in a way to make them more 
in control of how they process meaning and thus to develop 
beliefs and behaviors that serve their own needs better. 
Therefore, the effectiveness of interventions that are de¬ 
signed to achieve empowerment look for evidence of indi¬ 
viduals thinking for themselves, which suggests divergence 
much more than convergence. Thus, a media literacy inter¬ 
vention is successful with a range of opinions exhibited 
across participants as long as participants have developed a 
higher awareness of the process of developing their opinions 
and are able to support those opinions with evidence ar¬ 
ranged in logical arguments. This distinction can be suc¬ 
cinctly stated as the persuasion perspective focuses on giving 
people what researchers regard as a better fish, while the 
empowerment perspective focuses on teaching people how 
to fish better. 
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In our analysis, we frequently found a disconnect be¬ 
tween conceptual foundations and design decisions on this 
point. That is, frequently authors articulated media literacy 
by using empowerment-type language, such as attempting 
to get participants to think more critically. These researchers 
claimed that their media literacy interventions were designed 
to teach targets to make counter-readings of messages, resist 
the persuasive appeals of messages, and apply their own 
standards in formulating their own attitudes rather than 
accepting the attitudes and beliefs of product spokespersons, 
newscasters, and characters in fictional narratives. However, 
then these same authors exhibited design decisions that 
formulated a traditional persuasion study where they used 
the intervention to persuade participants to change their 
beliefs or behaviors. 

We are not arguing that empowerment type studies are 
inherently superior to persuasion type studies. Instead, we 
argue that designers of media literacy intervention studies 
need to be clear about which perspective they favor then 
design an appropriate study to achieve their intention. There 
is room under the broad media literacy umbrella for both 
types of studies. However, when researchers argue that their 
belief is empowerment, but then design a persuasion study, 
they create a hybrid that serves neither purpose well. 

Skills as performance or belief? Almost every conceptualiza¬ 
tion of media literacy suggests a skills component. Also, a 
high percentage of empirical tests of media literacy claim to 
deal with at least one skill. Recall that in our study, we found 
that among the 66 published studies that provided a media 
literacy conceptual foundation, 59 defined media literacy at 
least in part as requiring the development of skills. When 
designers of media literacy intervention studies confront the 
issue of skills, they must ask themselves whether they regard 
skills as performances or as people’s beliefs about abilities? 
The way researchers answer this question indicates how 
much of a challenge they are willing to undertake when 
designing their interventions and measures of media literacy 
skills. If researchers regard skills as beliefs about abilities, 
the challenge is relatively easy to meet. However, if research¬ 
ers regard skills as performances that involve the application 
of thinking processes to complete tasks, then the challenge 
is considerably higher. This increase in design complexity 
and cost is a likely reason why so few studies in our sample, 
treated skills as performances. 

We argue that there is value in measuring participants’ 
beliefs about their skills. The large research literature stimu¬ 


lated by social cognitive theory (Bandura, 2009) clearly 
shows that when people have strong efficacy beliefs, they are 
able to learn more and to use that learning. However, we also 
argue that there is a difference between people’s beliefs about 
their skills and their actual level of performance on those 
skills. Our reading of this media literacy intervention litera¬ 
ture has led us to infer that most authors who talk about the 
importance of skills to media literacy are referring to people’s 
abilities (a) to construct counter-readings of the surface 
meanings presented in messages, (b) to analyze those mes¬ 
sages in various contexts, (c) to infer motives of senders, and 
even (d) to construct their own messages. The above listed 
skills require assessments of individual’s performances in 
order to be able to plot those individuals on a range of abili¬ 
ties to perform those tasks. Moreover, skills require perfor¬ 
mance both in the intervention as well as in the measurement 
of them. In the athletic realm, basketball coaches do not ask 
prospective players: How well do you shoot free throws? 
(very good, good, average, below average). Instead, they 
observe their performance. Of course, beliefs are also impor¬ 
tant in the sense that if people believe they have good bas¬ 
ketball skills, they are likely to continue playing the game 
and work to improve their skills compared to people who do 
not hold such positive beliefs. However, it is a person’s ac¬ 
tual performance level more so than their beliefs that reflect 
how well the game is played. 

While determining the level of basketball players’ free 
throw skill through performance is relatively easy, determin¬ 
ing the level of media literacy skills is much more challeng¬ 
ing. Research designers can begin working on this challenge 
by using a three-step procedure. First, they need to clarify 
as much as possible what the skill is that they want to teach 
or improve. Second, they need to think about what the vari¬ 
ous levels of performance are on the skill are, then determine 
what observables would indicate performance at each level. 
And third, they need to think about the skill as requiring a 
sequence of sub-tasks, then design measures to track par¬ 
ticipants through the process of applying that skill in the 
completion of each step in the process. 

In summary, scholars who present definitions of media 
literacy that include a skills component need to think 
through the concerns presented in this section. They need 
to provide more detail in the form of specifying domains of 
skills and be more clear about defining what those skills are. 
They also need to articulate their vision about whether they 
are dealing with skills or competencies and if skills, then are 
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those skills broad or specialized to media literacy. Scholars 
who clearly lay out their positions on these issues will be 
providing a great deal more guidance to designers of mea¬ 
sures. Then the designers of those measures need to be more 
complete and accurate in their operationalizations in order 
to achieve validity. 

VI. Conclusion 

This study has presented an examination of the validity 
of 88 published studies of media literacy interventions. 
While this literature has been generated by sincere scholars 
who have invested a great deal of effort designing interven¬ 
tions trying to help people improve their media literacy, this 
analysis revealed that the media literacy intervention litera¬ 


ture generally exhibits a low level of scholarly quality in the 
presentation of meaning as well as incomplete and inaccu¬ 
rate follow through on the design of measures. Thus, most 
of the studies in this literature have failed to establish a 
minimum level of validity. 

Our analysis also illuminated some important issues for 
designers of media literacy intervention studies to consider 
in their designs, including the basic criterion of presenting a 
definition for media literacy, whether the study being con¬ 
ducted is truly an evaluation of an intervention or a basic 
research study, the purpose of media literacy interventions, 
and how skills should be treated. If we wish to increase the 
validity of our media literacy intervention studies, we must 
clearly articulate what we mean by media literacy and use 
those meanings to guide our research design decisions in a 
much more systematic manner. 
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Table 1. Sampling of Definitions of Media Literacy, (back to pg. 3) 


From Scholars 

Adams & Hamm (2001): “the ability to create personal meaning from the visual and verbal symbols we take in every day 
from television, advertising, film, and digital media. It is more than inviting students to simply decode information. They 
must be critical thinkers who can understand and produce in the media culture swirling around them” (p. 33). 

Anderson (1981): the “skillful collection, interpretation, testing and application of information regardless of medium or 
presentation for some purposeful action (p. 22). 

Naiditch (2013): “the ability to develop and use critical thinking skills (such as sorting through, analyzing, and assessing 
information) to interpret media messages and to create meanings out of those messages...by becoming media literate, 
people learn to use critical lenses both as consumers of media messages and as producers of their own messages”(p. 339) 

Scharrer & Raring (2012): “an inquiry-based process of consideration of multiple layers of a topic in the formation of one’s 
own evaluation” (p. 352). 

Sholle & Denski (1995): “it must be conceived as a political, social and cultural practice” (p. 17). 

Silverblatt & Eliceiri (1997): “a critical-thinking skill that enables audiences to decipher the information that they receive 
through the channels of mass communications and empowers them to develop independent judgments about media 
content” (p. 48). 


From Professional Associations 

National Communication Association: A media literate person understands how words, images, and sounds influence the 
way meanings are created and shared in contemporary society in ways that are both subtle and profound. A media liter¬ 
ate person is equipped to assign value, worth and meaning to media use and media messages, (http://www.natcom.org/ 
uploadedFiles/About_NCA/Leadership_and_Governance/Public_Policy_Platform/K-12Standards.pdf) 

National Council for the Social Studies: The multimedia age requires new skills for accessing, analyzing, evaluating, creat¬ 
ing, and distributing messages within a digital, global, and democratic society. The acquisition and application of critical 
analysis and media production skills are part of what constitutes media literacy, (http://www.socialstudies.org/positions/ 
medialiteracy) 

National Leadership Conference on Media: The ability to access, analyze, evaluate, and communicate messages in a wide 
variety of forms literacy. (Aufderheide, 1997). 

From Citizen Action Groups 

Action Coalition for Media Education: Encourage critical thinking and free expression, examine the corporate media system, 
and inspire active participation in society, (http://www.acmecoalition.org/) 

Center for Media Literacy: “A framework for accessing, analyzing, evaluating and creating media. The development of 
critical thinking and production skills needed to live fully in the 21st century media culture. Also defined as a “new vi¬ 
sion of literacy for the 21st century: the ability to communicate competently in all media forms, print and electronic, as 
well as to access, understand, and analyze and evaluate the powerful images, words and sounds that make up our con¬ 
temporary mass media culture.” Also, “Through a four-step ‘inquiry’ process of awareness. . .Analysis. . .Reflection. . 
.Action, media literacy helps young people acquire an empowering set of ‘navigational’ skills” which include the ability 
to access, analyze, evaluate, and create media. 

(http://www.medialit.org/about-cml) 

Citizens for Media Literacy: How to think critically about TV and advertising. Special emphasis is placed on the benefits 
of telling one’s own stories rather than being pre-occupied with manufactured stories designed to promote the purchase 
of products, (http://www.main.nc.us/cml/) 

Coalition for Quality Children’s Media (KIDS FIRST!): Teaching kids to become more critical media users and to reduce 
the impact of and exposure to violent and biased media. We teach them to recognize programs that are intellectually and 
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creatively stimulating; that break down racial, gender, handicapped and cultural boundaries; and that are produced with 
high technical and artistic standards, (http://www.kidsfirst.org/kidsfirst/html/whatcq.htm) 

Common Sense Media: The ability to identify, find, evaluate, and use information effectively. From effective search strate¬ 
gies to evaluation techniques, students learn how to evaluate the quality, credibility, and validity of websites, and give 
proper credit. 

(https://www.commonsensemedia.org/educators/digital-citizenship/information-literacy) 

Media Watch: challenge abusive stereotypes and other biased images commonly found in the media, (http://www.medi- 
awatch.com/?page_id=32) 

National Association for Media Literacy Education: The purpose of media literacy education is to help individuals of all 
ages develop the habits of inquiry and skills of expression that they need to be critical thinkers, effective communicators 
and active citizens in today’s world, (https://namle.net/publications/core-principles/) 

Governmental Groups 

Eunice Kennedy Shriver National Institute of Child Health and Human Development (through the Media Smart Program): 
having awareness of and critical thinking skills about the media’s role in influencing choices related to nutrition and 
physical activity. Young people should learn to question the who, what, why, and how behind words and images in the 
media. As a result, they develop critical thinking skills that help them form their own opinions and make their own 
choices about the messages they see and hear. (https://www.nichd.nih.gov/msy/Pages/index.aspx) 

New Mexico Media Literacy Project (now known as the Media Literacy Project): the ability to critically consume and create 
media. Media literate individuals are better able to decipher the complex messages they receive from television, radio, 
newspapers, magazines, books, billboards and signs, packaging and marketing materials, video games, and the Internet. 
Media literacy skills can help one understand not only the surface content of media messages but the deeper and often 
more important meaning beneath the surface, (https://medialiteracyproject.org/learn/media-literacy/) 

Office of National Drug Control Policy: “To a) recognize how media messages influence us (e.g. develop a vocabulary to 
recognize manipulative techniques, develop skills to protect oneself against messages about drugs or negative lifestyle 
choice that are embedded in the media), to b) develop critical thinking (e.g. know that messages are constructed by 
people with points of view and commercial interests, uncover value messages inherent in media, evaluate information 
for accuracy and reliability), to foster self-esteem (e.g., creatively produce satisfying and constructive messages)”. (Levitt 
& Denniston, 2014; https://www.ncjrs.gov/ondcppubs/pdf/strat_ptl.pdf) 
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Table 2. Definitions for Media Literacy: Components and Domains, (back to pg. 3) 


Skills Focused Components 


Generic Skills 

Skill building (Alliance for a Media Literate America) 

Skills necessary for competent participation in communication across various types of electronic audio and visual media” 
(Speech Communication Association, 1996, Standard 23) 

Skills of Accessing 

Ability to access media messages (Media Literacy Project, n.d.) 

Ability to access meaning from media messages (Adams & Hamm, 2001; Anderson, 1981; Silverblatt & Eliceiri, 1997) 
Skills of Interpretation 

Ability to make one’s own interpretations from media messages (Anderson, 1981; Adams & Hamm, 2001; Silverblatt & 
Eliceiri, 1997) 

Ability to use aesthetic building blocks to create and shape cognitive and affective mental maps (Zettl, 1998) 

Ability to analyze media messages (Anderson, 1981; Adams & Hamm, 2001; Brown, 1998) 

* Particularly ideological analysis, autobiographical analysis, nonverbal communication analysis, mythic analysis, and 
analysis of production techniques (Silverblatt, Ferry, & Finan, 2015) 

* Critical thinking about media messages (Adams & Hamm, 2001) 

* “a critical-thinking skill that enables audiences to decipher the information that they receive through the channels of 
mass communications and empowers them to develop independent judgments about media content” (Silverblatt & Eliceiri, 
1997, p. 48). 

Skills of Message Production 

Ability to communicate effectively by writing well (Brown, 1998) 

Ability to produce media messages (Adams & Hamm, 2001; Auferheide, 1997 Hobbs, 1998 
Ability to create counter-representations of media messages (Sholle & Denski, 1995) 

Knowledge Components 

Knowledge of Media Industry 

“knowledge about how the mass media function in society. . . Ideally, this knowledge should encompass all aspects of the 
workings of the media: their economic foundations, organizational structures, psychological effects, social consequenc¬ 
es, and, above all, their ‘language,’ that is the representational conventions and rhetorical strategies of ads, TV programs, 
movies, and other forms of mass media content” (p. 70); “an understanding of the representational conventions through 
which the users of media create and share meanings” especially visual representations. (Messaris, 1998, p. 70) 
Understanding the process, context, structure, and production values of the media (Silverblatt, 1995) 

Knowledge of Media Content 

Understanding of media content (understanding of the conduits that hold and send messages), of media grammar (under¬ 
standing of the language or aesthetics of each medium), and of the medium (understanding of the type of setting or en¬ 
vironment) (Meyrowitz, 1998) 

Knowledge of Media Effects 

Understand the effects of the various types of electronic audio and visual media, including television, radio, the telephone, 
the Internet, computers, electronic conferencing, and film, on media consumers.” (Speech Communication Association, 
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1996, Standard 22) 

Understanding of how the media distort aspects of reality as they manufacture their messages and how symbol systems 
mediate our knowledge of the world (Masterman, 1985) 

Learning about “text processing within the broad and complex context of a social, cultural, educational, and commercial 
textual ecosphere” (Mackey, 2002, p. 8) 

Understanding how media messages shape people’s construction of knowledge of the world and the various social, eco¬ 
nomic, and political position they occupy within it” (Alvermann, Moon, & Hagood, 1999, pp. 1 - 2) 

Knowledge about One’s Self 

Understanding of one’s place in the world (Blanchard & Christ, 1993; Sholle & Denski, 1995) 

Behavioral Components 

Generic 

“a political, social and cultural practice” (Sholle & Denski, 1995, p. 17) 

Empowerment 

Becoming empowered citizens and consumers (Blanchard & Christ, 1993; McLaren, Hammer, Sholle, & Reilly, 1995; Sholle 
& Denski, 1994) 

Moving people from dependency to self direction by being more reflective (Grow, 1990) 

“Policing one’s own viewing behaviour - if not by reducing the amount of television they watch, then at least by watching 
it in ways which are assumed to minimize its influence” (Buckingham, 1993, p. 21) 

Becoming sophisticated citizens rather than sophisticated consumers (Lewis & Jhally, 1998) 

Empowering and liberating people to prepare them for democratic citizenship and political awareness” (Masterman, 1985, 
p. 15, writing about the Council of Europe Resolution on Education in Media and New Technologies which was ad¬ 
opted by European Ministers of Education). 

Activism 

Becoming stimulated by social issues that are influenced by the media; these issues are things like violence, materialism, 
nutrition, body image, distortion in news reporting, and stereotyping by race, class, gender, and sexual orientation 
(Anderson, 1983) 

Becoming “active, free participants in the process rather than static, passive, and subservient to the images and values com¬ 
municated in a one-way flow from media sources” (Brown, 1998, p. 47) 

Challenging abusive stereotypes and other biased images commonly found in the media (Media Watch) 

Social Networking 

Creating communities of people who interact in complex social and cultural contexts and use this awareness to decide what 
textual positions to accept (Buckingham, 1998) 

“primarily something people do; it is an activity, located in the space between thought and text. Literacy does not just reside 
in people’s heads as a set of skills to be learned, and it does not just reside on paper, captured as texts to be analysed. Like 
all human activity, literacy is essentially social, and it is located in the interaction between people” (Barton & Hamilton, 
1998, p. 3; cited in Margaret Mackey, 2002, p. 5-6) 

Affective Components 

Pay more attention to one’s own affective investment as one consumes the media (Sholle & Denski, 1995) 

Ability to appreciate media messages (Adams & Hamm, 2001) especially respected works of literature (Brown, 1998) 
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